Towards Real Time Discovery from Distributed Information Sources

Many successful knowledge discovery or data mining techniques and systems have been developed. These techniques usually apply to centralized databases with less restricted requirements on learning and response time. Not so much effort yet has been put into mining of distributed databases and real-time issues. In this paper, we investigate issues of fast distributed data mining. We assume that merging the distributed database into a single one would be either be too costly (distributed case); or, the individual fragments are non-uniform so that mining only one fragment would bias the result (fragmented case). The goal is to classify objects O of the database into one of several mutually exclusive classes Ci. Our approach to make mining fast and feasible is as follows. From each data site or fragment dbk, only a single rule rik is generated for each category Ci. A small subset {ri1, ...,rih} of these individual rules is selected to form a rule set Ri for each category Ci. These rule subsets represent adequately the hidden knowledge of the entire database. Various selection criteria to form Ri are discussed, both theoretically as well as experimentally. Each rule set Ri represents one category Ci saying whether an object O belongs to Ci or not. However, it may still be that O is classified by Ri and Rj as belonging to Ci and Cj respectively which is not possible since the classes are mutually exclusive. Various novel methods of finally making the classifications mutually exclusive are investigated. A superior such method is identified.